-
Notifications
You must be signed in to change notification settings - Fork 426
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
chore(langchain): auto-instrument with langgraph #12208
Conversation
|
BenchmarksBenchmark execution time: 2025-02-12 20:42:40 Comparing candidate commit fe66852 in PR branch Found 0 performance improvements and 1 performance regressions! Performance is the same for 393 metrics, 2 unstable metrics. scenario:iast_aspects-ospathdirname_aspect
|
Datadog ReportBranch report: ✅ 0 Failed, 130 Passed, 1378 Skipped, 4m 44.97s Total duration (35m 12.56s time saved) |
5696d35
to
ad56495
Compare
52e9c96
to
3f08852
Compare
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Some initial thoughts
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Needs a release note with a feature and two fixes. The change itself looks fine.
Adds gated auto-instrumentation for LangChain with LangGraph, along with a couple small fixes for LangGraph. ### LangGraph bug fixes 1. We were not properly recording errors from LangGraph spans, and not setting LLMObs tags in the case of failures. We add our `set_llmobs_tags` in the appropriate spots to resolve this. 2. LangGraph will sometimes have triggers for a step be something like `["__pregel_push"]` in the case when a `Send` is enqueued as a trigger, and will not give hints to the actual invoker. To resolve this, we check that if for any given queued task, there is only one finished task. If so, set that task as the trigger. ### Complications with LangChain linking The primary obstacle was that, unlike with LangGraph, where we had an intermediary function to patch between tasks executed by the graph, we don't have that for LangChain LCEL chains. Their steps are executed in a loop inside their `invoke` method, which blocks us from jumping in between the steps to make the links. Additionally, LangChain elements that we trace are sometimes embedded in other `Runnable` types: - `RunnableBinding`, which binds an instance inside of it - `RunnableAssign`, which embeds an instance of a `RunnableParallel` - `RunnableParallel`, which can run multiple `Runnables` (some of which we might trace) in parallel, which have the parallel items in a `steps__` attribute Something I tried to overcome this was to flatten/flatmap the items of the list of steps to extract these. To overcome linking between steps, I recorded the instance of each traced Runnable, mapping its ID to its span, and vice versa, to be able to grab instances if needed. Additionally, for each Runnable item in the chain, I marked them as a chain step by adding them to a set of steps to later check against. ### Setting links Link setting is split into setting the input links (`"to": "input"`) and output links (`"to": "output"`) #### Input Links Input links are set by: 1. Identifying if the span represents a step in a chain. If **not**, set its input link as `input --> input` from the parent span. 3. If it does represent a step in the chain, find the index of the previous traced step in the chain (the chain instance is grabbed from the span to instance mapping referenced above), and setting it as the `output --> input` link. If the step contains multiple spans (ie from a `RunnableParallel`), add all of those spans as links with the same `output --> input` attribute The index of the previously traced step in the chain (`-1` if not found or is not a chain step) is returned for use in output linkage. #### Output Links Output links are set by: 1. If the span does not represent a step in a chain, or the parent span isn't a chain (ie has a `steps` attribute), set the `output --> output` link from the current span onto the parent span. 2. If the span does represent a step a chain, remove all previous span links on the span from the previous traced step, and set the new span link from the current span. We do this overwriting every time because we don't know ahead of time which step in the chain will be the last one we trace, so we have to remove previous span links if we find we need to add a new one. ## Checklist - [x] PR author has checked that all the criteria below are met - The PR description includes an overview of the change - The PR description articulates the motivation for the change - The change includes tests OR the PR description describes a testing strategy - The PR description notes risks associated with the change, if any - Newly-added code is easy to change - The change follows the [library release note guidelines](https://ddtrace.readthedocs.io/en/stable/releasenotes.html) - The change includes or references documentation updates if necessary - Backport labels are set (if [applicable](https://ddtrace.readthedocs.io/en/latest/contributing.html#backporting)) ## Reviewer Checklist - [x] Reviewer has checked that all the criteria below are met - Title is accurate - All changes are related to the pull request's stated goal - Avoids breaking [API](https://ddtrace.readthedocs.io/en/stable/versioning.html#interfaces) changes - Testing strategy adequately addresses listed risks - Newly-added code is easy to change - Release note makes sense to a user of the library - If necessary, author has acknowledged and discussed the performance implications of this PR as reported in the benchmarks PR comment - Backport labels are set in a manner that is consistent with the [release branch maintenance policy](https://ddtrace.readthedocs.io/en/latest/contributing.html#backporting)
Adds gated auto-instrumentation for LangChain with LangGraph, along with a couple small fixes for LangGraph. ### LangGraph bug fixes 1. We were not properly recording errors from LangGraph spans, and not setting LLMObs tags in the case of failures. We add our `set_llmobs_tags` in the appropriate spots to resolve this. 2. LangGraph will sometimes have triggers for a step be something like `["__pregel_push"]` in the case when a `Send` is enqueued as a trigger, and will not give hints to the actual invoker. To resolve this, we check that if for any given queued task, there is only one finished task. If so, set that task as the trigger. ### Complications with LangChain linking The primary obstacle was that, unlike with LangGraph, where we had an intermediary function to patch between tasks executed by the graph, we don't have that for LangChain LCEL chains. Their steps are executed in a loop inside their `invoke` method, which blocks us from jumping in between the steps to make the links. Additionally, LangChain elements that we trace are sometimes embedded in other `Runnable` types: - `RunnableBinding`, which binds an instance inside of it - `RunnableAssign`, which embeds an instance of a `RunnableParallel` - `RunnableParallel`, which can run multiple `Runnables` (some of which we might trace) in parallel, which have the parallel items in a `steps__` attribute Something I tried to overcome this was to flatten/flatmap the items of the list of steps to extract these. To overcome linking between steps, I recorded the instance of each traced Runnable, mapping its ID to its span, and vice versa, to be able to grab instances if needed. Additionally, for each Runnable item in the chain, I marked them as a chain step by adding them to a set of steps to later check against. ### Setting links Link setting is split into setting the input links (`"to": "input"`) and output links (`"to": "output"`) #### Input Links Input links are set by: 1. Identifying if the span represents a step in a chain. If **not**, set its input link as `input --> input` from the parent span. 3. If it does represent a step in the chain, find the index of the previous traced step in the chain (the chain instance is grabbed from the span to instance mapping referenced above), and setting it as the `output --> input` link. If the step contains multiple spans (ie from a `RunnableParallel`), add all of those spans as links with the same `output --> input` attribute The index of the previously traced step in the chain (`-1` if not found or is not a chain step) is returned for use in output linkage. #### Output Links Output links are set by: 1. If the span does not represent a step in a chain, or the parent span isn't a chain (ie has a `steps` attribute), set the `output --> output` link from the current span onto the parent span. 2. If the span does represent a step a chain, remove all previous span links on the span from the previous traced step, and set the new span link from the current span. We do this overwriting every time because we don't know ahead of time which step in the chain will be the last one we trace, so we have to remove previous span links if we find we need to add a new one. ## Checklist - [x] PR author has checked that all the criteria below are met - The PR description includes an overview of the change - The PR description articulates the motivation for the change - The change includes tests OR the PR description describes a testing strategy - The PR description notes risks associated with the change, if any - Newly-added code is easy to change - The change follows the [library release note guidelines](https://ddtrace.readthedocs.io/en/stable/releasenotes.html) - The change includes or references documentation updates if necessary - Backport labels are set (if [applicable](https://ddtrace.readthedocs.io/en/latest/contributing.html#backporting)) ## Reviewer Checklist - [x] Reviewer has checked that all the criteria below are met - Title is accurate - All changes are related to the pull request's stated goal - Avoids breaking [API](https://ddtrace.readthedocs.io/en/stable/versioning.html#interfaces) changes - Testing strategy adequately addresses listed risks - Newly-added code is easy to change - Release note makes sense to a user of the library - If necessary, author has acknowledged and discussed the performance implications of this PR as reported in the benchmarks PR comment - Backport labels are set in a manner that is consistent with the [release branch maintenance policy](https://ddtrace.readthedocs.io/en/latest/contributing.html#backporting)
Adds gated auto-instrumentation for LangChain with LangGraph, along with a couple small fixes for LangGraph.
LangGraph bug fixes
set_llmobs_tags
in the appropriate spots to resolve this.["__pregel_push"]
in the case when aSend
is enqueued as a trigger, and will not give hints to the actual invoker. To resolve this, we check that if for any given queued task, there is only one finished task. If so, set that task as the trigger.Complications with LangChain linking
The primary obstacle was that, unlike with LangGraph, where we had an intermediary function to patch between tasks executed by the graph, we don't have that for LangChain LCEL chains. Their steps are executed in a loop inside their
invoke
method, which blocks us from jumping in between the steps to make the links.Additionally, LangChain elements that we trace are sometimes embedded in other
Runnable
types:RunnableBinding
, which binds an instance inside of itRunnableAssign
, which embeds an instance of aRunnableParallel
RunnableParallel
, which can run multipleRunnables
(some of which we might trace) in parallel, which have the parallel items in asteps__
attributeSomething I tried to overcome this was to flatten/flatmap the items of the list of steps to extract these.
To overcome linking between steps, I recorded the instance of each traced Runnable, mapping its ID to its span, and vice versa, to be able to grab instances if needed. Additionally, for each Runnable item in the chain, I marked them as a chain step by adding them to a set of steps to later check against.
Setting links
Link setting is split into setting the input links (
"to": "input"
) and output links ("to": "output"
)Input Links
Input links are set by:
input --> input
from the parent span.output --> input
link. If the step contains multiple spans (ie from aRunnableParallel
), add all of those spans as links with the sameoutput --> input
attributeThe index of the previously traced step in the chain (
-1
if not found or is not a chain step) is returned for use in output linkage.Output Links
Output links are set by:
steps
attribute), set theoutput --> output
link from the current span onto the parent span.Checklist
Reviewer Checklist